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The set of the fixed points of the Hopfield type network is under investigation. The connection 
matrix of the network is constructed according the Hebb rule from the set of memorized patterns 
0^ , which are treated as distorted copies of the standard-vector. It is found that the dependence of the 

ON . set of the fixed points on the value of the distortion parameter can be described analytically. The 

obtained results are interpreted in the terms of neural networks and the Ising model. 
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£T) I 1°. The problem of maximization of a symmetric form which is quadratic in spin variables <Ji\ 
<N ■ 

f F(#) = Ji 3 OiVj -> max, o l = {±1}, 

\ a= (o"i, . . . ,a n ), Jij = Jji, i,j = 1,2, ...,n. 

is under investigation. This problem arises in the Ising model, in the surface physic^ iri the theory of optimal coding, 
in factor analysis, in the theory of neural networks and in the optimization theoryEnl Here the aim is to obtain an 
effective method for the search of the global maximum of the functional and a constructive description of the set of 
j • I its local extrema. 

The rt-dimcnsional vectors a, which define 2" configurations, will be called the configuration vectors. The configu- 
ration vector which gives the solution of the problem (1) will be called the ground state. We investigate the problem 
(1) in the case of the connection matrix constructed with regard to the Hebb rule from the (p x n)-matrix S of the 
' O ■ form 



O 

o 



I l-x 1 ... 1 1 ... 1 \ 



1 l-x ... 1 1 ... 1 



\ 1 1 ... 1 -x 1 ... 1 / 



(2) 



iy) ' where x is an arbitrary real number. We introduce the special notation N for the related connection matrix: 

N = S T -S, N«=0, i=l,2,...,n. (3) 

o ■ , n 

Q\ ' According to the conventional neural network tradition we treat the n-dimensional vectors > , which are the rows 
0^ ■ of the matrix S, as p memorized patterns embedded in the network memory (it does not matter that not all the 
elements of the matrix S are equal {±1}; see Note 1). Then the following meaningful interpretation of the problem 
can be suggested: the network had to be learned by p-time showing of the standard e(n) — (1, 1, ... , 1), but an error 
had crept into the learning process and in fact the network was learned with the help of its p distorted copies; the value 
of the distortion x was the same for all the memorized patterns and every time only one coordinate had been distorted: 



I 



^ = (l,...,l,l-z,l,...,l), 1 = 1,2,. ..,p. (4) 



When x is equal zero, the network is found to be learned by p copies of the standard s[n). It is well-known that 
in this case the vector e{n) itself is the ground state and the functional has no other local maxima. For continuity 
' reasons, it is clear that the same situation remains for a sufficiently small distortion x. But when x increases the 
I ground state changes. For the problem (l)-(3) we succeeded in obtaining the analytical description of the dependence 
of the ground state on the value of the distortion parameter. 

Notations. We denote by e(k) the configuration vector which is collinear to the bisectrix of the principle orthant 
of the space R k . The vector, which after p distortions generates the set of the memorized patterns s^, is called the 
standard-vector. Next, n is the number of the spin variables, p is the number of the memorized patterns and q — n—p 
is the number of the nondistorted coordinates of the standard-vector. Configuration vectors are denoted by small 
Greek letters. We use small Latin letters to denote vectors whose coordinates are real. 

Note 1. In the neural network theory the connection matrix obtained with the help of Eq.(3) from (p x n)-matrix 
S whose elements are equal {±1} is called the Hebb matrix. If in Eq.(3) (p x n)-matrix S is of the general type, the 
corresponding connection matrix will be called the matrix of the Hebb type. 
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With regard to the set of the fixed points of the network an arbitrary symmetric connection matrix with zero 
diagonal elements is equivalent to a matrix of the Hebb type. Indeed, equality of the diagonal to zero guarantees the 
coincidence of the set of the local maxima of the functional (1) with the set of the network's fixed points. But the 
local maxima do not depend on the diagonal elements of the connection matrix, so the lasts can be chosen whatever 
we likeu. In particular, all the diagonal elements can be taken so large that the connection matrix becomes a positive 
definite one. And such a matrix can be already presented in the form of the matrix product (3), where the elements of 
the related matrix S are not necessarily equal to {±1}. In other words, with the help of the simple deformation of the 
diagonal a symmetric connection matrix turns into the Hebb type matrix and as a result, the set of the local maxima 
of the functional (1) does not change. This reasoning is correct for the Hebb matrix too, since it is a symmetric 
one and its diagonal elements are equal zero. In such a way we ascertain the actuality of the Hebb type connection 
matrices for the Hopfield model (for details seeta). 

2°. Basic model. Let us look for the local maxima among configuration vectors whose last coordinate is positive. 
Since q last columns of thpjnatrix S are the same, the configuration vector which is "under the suspicion" to provide 
an extremum is of the formE£!l 



(ci, (T 2 , . . . , cr p , 1, . . . , 1), 



(•5) 



where we denote by. a' the p-dimension part of the vector a*, which is formed by its first p coordinates. The direct 
calculations (or seeo) show that 



where 



F(a*) oc x 2 — 2x(q -fpcosw) cos it; + (q + pcosw) 2 , (6) 



i=l (Ji r*7\ 
cos w = — — (7) 



is the cosine of the angle between the vectors a 1 and s(p). Depending on the number of the coordinates of the vector 
a' whose value is "— 1", cosu; takes the values cos Wk = 1 — 2k /p, where k = 0, 1, . . . ,p. Consequently, 2 P "suspicious- 
looking" vectors a* are grouped into the p+ 1 classes E^: the functional F(a*) has the same value Fk(x) for all the 
vectors from the same class. The classes E& are numerated by the number k of the negative coordinates which have 
the relevant vectors 5* , and the number of the vectors in the k- class is equal to Cp. 

To find the ground state under a given value of x, it is necessary to determine the greatest of the values 
Fq(x), Fi(x), . . . , F p (x). Under the comparison the term x 2 can be omitted. Therefore, to find out how the ground 
state depends on the parameter x, it is necessary to examine the family of the straight lines 

L>k(x) = (q + pcoswk) 2 — 2x(q + poos wk) cos wk '■ (8) 

in the region where the Lk(x) majorizes all the other straight lines, the ground state belongs to the class E& and is 
Cp times degenerated. The analysis of the relative position of the straight lines Lk(x) givesQ: 

Theorem. As x varies from — oo to oo the ground state in consecutive order belongs to the classes Eg, Si, ... , Efc max . 
The jump of the ground state from the class T^k-i into the class S& occurs at the point Xk of intersection of the straight 
lines Lk-\(x) and Lk(x): 

n- (2k- 1) 
Xk - p n+p-2(2k-iy fc - 1 ' 2 '---' fc ™— 

V n~^T ^ 5' one a ft er another all the p rebuildings of the ground state take place according the above scheme: k max — p. 
And if > ^, the last rebuilding is the one whose number is k max = [ n+ ^ ] . The functional has no other local 
maxima. 

This theorem allows to solve a lot of practical problems. 

In Fig.l the typical examples of the relative position of the straight lines Lk(x) are presented for the cases < | 
(a) and > § 3 )- When x changes from — oo to xi, the ground state is the standard- vector e{n) (it exhausts 
the class So). I n the point x\ the ground state jumps from the class Eo to the class Ei and becomes p times 
degenerated. When x achieves the value X2, the ground state jumps from the class Ei to the class E 2 and becomes C 2 
times degenerated, and so on. As x increases, the value of the functional for the ground state at first monotonically 
decreases and then, after reaching the minimum value, increases monotonically. For what follows let us note that 
kmax > t 2 ^], and x kmaw > p. 
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a). b). 

FIG. 1. The typical behavior of the straight lines Lk(x), k = 0, 1, . . . ,p. The rebuildings of the ground state occurs at the 
points Xk of the intersection of the straight lines L^-\ and L^. On the interval (xt, x^+i) the ground state is the one with 
the number k. When x increases: a), all the rebuildings of the ground state occur (fc mal = 5), because < |i b). only 3 
rebuildings of the ground state occur {km ax = [ n+ ^ +2 ] = 3), because > |. 



The case p = n worth to be specially mentioned. Here all the jump points x^ stick to one point 

+ 1" 



I Tl 



(9) 



For any x from the left of x' the ground state is the standard- vector e(n) , and for x from the right of x' the ground 

[—1 

state belongs to the class Ep+i-, and is C„ 2 times degenerated. 

The interval X\ < x < £& max will be called the rebuilding region of the ground state. This region is examined in 
details irM Here we would like to mention only that its left boundary x\ > ^ is the monotonically increasing function 
of p as well as of n. And also, when p — const and n — > oo the rebuilding region tightens to the point 

x" = p. (10) 

Here for x < x" the ground state is the standard- vector and for x > x" the ground state belongs to the class E p ; 
again it is a nondcgcnerate one. _ 
Note 2. The Theorem remains valid, ifS: 

a) . The memorized patterns (4) are normalized to unit to prevent their length being dependent on the varying 
parameter x. As a result the maximum value of the functional (1) for the ground state decreases monotonically as 
function of x. 

b) . An arbitrary configuration vector a = (oti, ct2, ■ ■ ■ , u p , a p +i, . . . , a n ) is used in place of the standard- vector e{n). 
Then all the results are formulated with respect to configuration vectors a* = (a%ai, 0:202, • • ■ , ct v a v , a p +i, . . . , a n ), 
and the elements of the connection matrix are chanched: 

N^f =N ijaiaj , '../ I-'-' n. (11) 

c) . The first p coordinates of the space R n are subjected to a rotation. If in this connection the standard- vector does 
not change, all the results of the "Basic model" are valid, though in this case the memorized patterns are obtained 
from e[n) by simultaneous distortion of its p coordinates! But if as a result of the rotation the standard-vector turns 
into 

p 



(ui,u 2 , ■ • ■ ,u p , 1, . . . , 1), m £ R 1 , 2^uf=p, 



1=1 
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the elements of the relevant connection matrix take the form 



Nlp=N ijUiUj , i,j = 1,2,..., n. (12) 

Then, as in the "Basic model", the vectors a*, which are "under the suspicion" to provide an extremum, are grouped 
in the classes where the value of the functional is constant. But now the vectors a* belong to the same class if their 
p-dimensional parts are equidistant from the vector u = {u\,U2, ■ ■ ■ ,u p ). And just as before when x increases, the 
ground state in consecutive order jumps from a class to the next one. Here the Theorem remains valid, however a 
correction of the formulae is necessary. We would like to note, that since the choice of the values {ui}f =1 is completely 
in the researcher's hand, it is possible to construct the Hopfield type networks with preassigned sets of fixed points. 
An additional analysis is required to find out the limits of this method. 

We would like to mention that we succeeded in the generalization of the " Basic model" to the case when the linear 
term h X)"=i a i was added to the functional F(a) which had to be maximized. In physics models due to such a term 
the external magnetic field can be taken into account. Now we prepare these results for publication. 

3°. Interpretations. Let us in short discuss the results which are relative to the "Basic model". 

Neural networks. In this case the Theorem has to be interpreted in the framework of the meaningful setting of 
the problem, which has been done above (see 1°): the quality of "the truth" (the standard-vector) reconstruction by 
the network depends on the distortion value x during the learning stage and on the length p of the learning sequence. 

In agree with the common sense the error of the network increases with the increase of the distortion x. 

Also it is quite reasonable that the left boundary of the rebuilding region x\ is the increasing function of p and n. 
Indeed, when n and x are fixed, merely due to increase of the number of the memorized patterns p the value of x\ can 
be forced to exceed x (of course, if x is not too large). As a result x turns out to be left of x\, i.e. in the region where 
the only fixed point is the standard-vector. This conclusion is in agreement with the practical experience according 
which the greater the length of the learning sequence, the better the signal can be read through noise. In the same 
way the increasing of x\ with the increasing of the number n can be interpreted. 

When p = const and n — » oo all the jump points Xk stick to one point x" = p. In this case for x < x" the ground 
state is the vector which belongs to the class T, : 

£<+)(n) = (l,l,...,l,l,...,l), (13) 
» ' 

p 

and for x > x" the ground state is the vector which belongs to the class £ p : 

£<-)(n) = (-l,-l,...,-l,l,...,l) (14) 

S V ' 

p 

(see Eq.(lO)). As we see it, this result is a nontrivial one. In terms of the learning process, the distinct parts of the 
vectors £^ + ^(n) and e(~>(n) are two opposed statements. And the network "feels" this. When the distortions x is 
not very large (less than x") the memorized patterns s^> (4) are interpreted by the network as the distorted copies 
of the vector e^ + ^(n) (13). But if during the learning stage the distortions exceed x", the network interprets the 
memorized patterns as the distorted copies of other standard- vector '{n) (14). The last result is in agreement 
with the practical experience too: we interpret deviations in the image of a standard as permissible ones only till 
some threshold.^ only this threshold is exceeded, the distorted patterns are interpreted as quite different standard. 
(For details seeEI. In the same reference the very interesting dependence of k max on the relation between p and n is 
discussed.) 

The Ising model at T=0. The interpretation of this model in terms of the matrix S is not known yet. Therefore 
here the obtained results are interpreted starting from the form of the Hamiltonian N (3). Let's write it in the 
block-matrix form: 



Noc 



A B 
B T C 



where the diagonal elements of the (p x p)-matrix A and the (q x ^-matrix C are equal zero, and 

a l3 ■, = 1 - 2y, i, j = 1,2, . . . ,p, i + y, 

hk = l-y, i = 1, 2, . . . ,p, k=l,2,...,q; 

cm = 1, k, I = 1,2, . . . ,q, k^=l; 
y — -. 
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This matrix corresponds to a spin system with the infinitely large interaction radius. The system consists of two 
subsystems, which are homogeneous with regard to the spin interaction. The interaction between the p spins of the 
first subsystem is equal to 1 — 2y; the interaction between the q spins of the second subsystem is equal to 1; the 
crossinteraction between the spins of each subsystems is equal to 1 — y. When p = n all the spins are interacting with 
each other in the same way. (We would like to remind that the connection matrix can be generalized - see Eqs. (11), 

(12) 

While y < | all the spins are interacting in the ferromagnetic way; when | < y < 1, the interaction between the spins 
of the first subsystem becomes of antiferromagnetic type, and when 1 < y the crossinteraction is of antiferromagnetic 
type too. The Theorem allows to trace how the ground state depends on the variation of the parameter y. 

Let p < n. For y G (-co, |) the ground state is the ferromagnetic one since | < y\ — and for x < %\ the ground 

state is the standard-vector e^ + ^(n) (13). But it is interesting that the ground state remains the ferromagnetic one 
even if ^ < y < yi, i.e. when the antiferromagnetic interactions already shown up in the system. In other words, when 
p < n there is "a gap" between the value of the external parameter y which corresponds to the destruction of the 
ferromagnetic interaction and the value of this parameter which corresponds to the destruction of the ferromagnetic 
ground state. Only after a "sufficient amount" of the antiferromagnetic interactions is accumulated, the first jump 
of the ground state occurs and it ceases to be the ferromagnetic one. Then after another critical "portion" of the 
antiferromagnetic interaction is accumulated the next jump of the ground state occurs (it happens when y exceeds 
?/2 = — ), and so on. After the parameter y reaches the value y" = 1 = — , the crossinteraction becomes the 
antiferromagnetic one too. The ground state continues "to jump" after that, since Xk max > P- 

The energy E = — F of the ground state as a function of the parameter y has breaks at the points y^ — ^ . It 
increases till y < y" = 1 and decreases when y > y" . However, if the memorized patterns are normalized to unit, the 
energy of the ground state is a monotonically increasing function of the external parameter. 

It is natural to treat the case p = const, n — > oo as the case of an infinitely large sample with a few number of 
impurities. In this case all yu stick to the point y" (see Eq.(lO)). Depending only on the type of the crossinteraction 
between the impurities and the rest of the sample, the ground state is either the ferromagnetic one (the vector s^ + ^ (n) 

(13) ), or the spins of the impurities are directed in an opposite way with respect to the other spins of the sample (and 
the ground state is the vector )(n) (14)). 

Finally, let's discuss the case p = n. Then all yk stick to the point y' = h (see Eq.(9)). Here the destruction of 
the ferromagnetic interaction occurs simultaneously with the change of the ground state ( "the gap" disappears). As 
long as the interaction of the spins is ferromagnetic (y < i), the ground state is ferromagnetic too. But when the 

interaction of the spins becomes antiferromagnetic (y > ^), the ground state turns out to be C n 2 times degenerated. 
From the right of i it is natural to associate the state of the system with the spin glass phase. 
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